Schooling of light reflecting fish

One of the hallmarks of the collective movement of large schools of pelagic fish are waves of shimmering flashes that propagate across the school, usually following an attack by a predator. Such flashes arise when sunlight is reflected off the specular (mirror-like) skin that characterizes many pelagic fishes, where it is otherwise thought to offer a means for camouflage in open waters. While it has been suggested that these ‘shimmering waves’ are a visual manifestation of the synchronized escape response of the fish, the phenomenon has been regarded only as an artifact of esthetic curiosity. In this study we apply agent-based simulations and deep learning techniques to show that, in fact, shimmering waves contain information on the behavioral dynamics of the school. Our analyses are based on a model that combines basic rules of collective motion and the propagation of light beams in the ocean, as they hit and reflect off the moving fish. We use the resulting reflection patterns to infer the essential dynamics and inter-individual interactions which are necessary to generate shimmering waves. Moreover, we show that light flashes observed by the school members themselves may extend the range at which information can be communicated across the school. Assuming that fish pay heed to this information, for example by entering an apprehensive state of reduced response-time, our analysis suggests that it can speed up the propagation of information across the school. Further still, we use an artificial neural network to show that light flashes are, on their own, indicative of the state and dynamics of the school, and are sufficient to infer the direction of attack and the shape of the school with high accuracy.

In addition to the N fish, simulations may involve a single predator, which appears for a limited number of steps and moves along a fixed trajectory. Usually, the predator moves towards and into the school, typically faster than the agents' speed in the schooling state.
Velocity-updates depend on the current internal state ( ) as follows (see Fig 1 in the MS).
-Schooling: Agent velocity is determined similar to the usual three zones rules, e.g. Couzin et al. (2002), i.e. a weighted average of attraction to all neighbours within a given distance Ro, alignment to all neighbours within a distance Ra and repulsion from neighbours within a distance Rr, such that Rr<Ra<Ro. See ( ) describes several adjustments to the traditional motion rules as follows, where -The vector is a constant that describes a preferred direction, i.e. an agent "wants" to migrate in the direction m (see Table 1).
d ( ) is vector that moves the agents towards a desired water depth (value on the z-axis, arbitrarily set to zero), where 3 = (0,0,1) and [⋅]. is the z component of the vector. See Figure S2 Fig for an example for the trajectory of a single agent, correcting its depth using this approach.
w is an attraction to a desired value on the x-axis calculated in the same manner as d .
-( ) is a Gaussian noise term with zero mean and covariance matrix 2 3 . Here, 3 is the 3x3 identity matrix.
Finally, the new velocity is taken to be, where is an inertia parameter (default value 0.95), qualitatively modeling the friction of water.
-Evasive: If an agent in the schooling state gains a speed that is higher than the schooling speed limit | ( )| > max schooling (for example, because it encountered a predator or an informed individual − see copy-response section), it transitions to an evasive state. In this state, the update rules for the velocity are similar to the rules while schooling, but with different weights that give higher priority to cohesion and alignment over repulsion. The evasive state lasts until the fish slows down by friction and by averaging out their response with their schooling neighbors.
-direct-only: If an agent in the schooling or evasive states encounters the predator (i.e. the predator is within a distance ), then the agent transitions into the predator-response state. In this state, agents turn towards the opposite direction of the predator and start swimming at speed max evasive , -direct-and-copy: A copy-response behaviour describes a schooling agent that is not within sight range of a predator ( ), not in the middle of a sharp change of its trajectory (the angle between consecutive velocity vectors < 45 o ), and not in the evasive state, but has a neighboring agent (within radius ) which is making a sharp turn and swimming fast (above a threshold) will transition into the copy-response state. To be precise, an informed individual is a neighbor within a range that made the sharpest turn, given that the turning angle is larger than a threshold (~80 o in our implementation) and the neighbor's speed is high enough (0.5 of the maximum escape speed max evasive ). In this state, agents copy the velocity of the informed neighbor. In (the unlikely) case that there are more than one "sharpest turn" neighbors one of them is chosen arbitrarily. The rule for the velocity update is given by where, Here, latency is a parameter. This parameter is the only one which is modified by response to flashes, as explained below.

Roll angles:
As explained above, in order to track the angle in which incident light beams are reflected off agents, one needs to model the dynamics of a direction normal to the velocity, corresponding to the back of the fish. We denote this direction as ( ). In simulations, ( ) is given by where ( ) is the torsion of the trajectory curve of the agent, ( ) = ( − 1) + ( ) − 2 ( − 1) + ( − 2), and ra ( ) is a roll-alignment motion rule Similar to previous notation, ra ( ) is the set of neighbors of fish i up to a distance ra and r ( ) is added Gaussian noise with variance r 2 3 .

Optics: See Fig in the MS.
We assume a light source that consists of two components: 1. we define a light direction vector = − 3 = (0,0, −1) and 2. an angle representing a uniform light distribution around the light direction. The agents are considered as 2-sided planar mirrors whose normal direction ( ) is perpendicular to the velocity and back vectors, In order to determine if agent reflects light towards position in a given time , we define a vector in the direction of ( ) − .
Then, we define the reflected vector of ( ) from the plane represented by the normal ( ), and another vector pointing from the agent position towards , If the angle between ( ) and the vector ( ) is smaller than , then the agent reflects light towards . Denote by ( ) a Boolean variable describing if agent i reflects light towards position x, Since the model is discrete, it is possible that an agent will not reflect light both in time and in time + 1, but the trajectory between the two times will cross a point in which it will flash. To test whether an agent was flashing between the steps, we take as the plane normal defined by the reflected rays in times and − 1, and check the angle between this plane to ( ). If the projection is falling on the arc drawn by the trajectory between the two reflection rays, and the angle is smaller than , the fish is flashing. We consider the agent as flashing in time in such case. This continuous flashing (the flashing between steps) is given by

Detection of the flashes by the agents:
We add a new assumption that agents can detect flashes and that the accumulative effect of many flashes can affect the dynamics. We assume that if an agent sees many new light reflections within a single step, or many existing reflections are turned off, it assumes that these changes indicate instability in the motion of the group. As a precaution (a flash-response state) they lower the latency (so it will respond faster). In other words, the effect of flashes only changes the latency in the transition from schooling to evasive-copy behaviour.
The way we quantify the total number of changes in the flash is by summing the number of agents that changed the value of between time to time + 1. We defined a set where ⊕ denotes exclusive or. Therefore, fc ( ) is the set of agents which flashed towards agent i at time t, but not at time t+1, or the other way around -flashed towards agent i at time t+1, but not at time t . The flash change measure is simply ( ) = | fc ( )|. We defined a flash-change threshold (parameter) and implemented two possible responses to this flash change, both options assume that the fish is in schooling state: 1. Flash-direct: i.e. the agent turns away from the center of mass of the changed flashes, keeping its speed from time . The idea behind it is that the center of mass of the flash changes may indicate the location of the cause of the disturbance.

Flash latency:
An agent that is exposed to above-threshold flash changes, changes its latency to one. The idea is that the flash changes inform the fish that something may be wrong without giving precise information. Because of that, the response of the fish is to be ready to react faster and anticipate more specific information. Figure S1. The relative dimensions of the copy-response zone and the three social zones that give rise to schooling state. The agent (the black fish in the middle) modifies its motion based on its neighbors' velocities and locations within the Cohesion zone (the agent is attracted towards the center of the mass of the neighbors' position within this zone, which is typically the largest). The Alignment zone (the agent adjusts its direction towards the average direction of its neighbors within this zone) and the Repulsion zone (the agent tries to get away of the agents in this zone by swimming away of their center of mass weighted by their relative distance from the agents' position. This zone, out of all zones, is typically the smallest and of the highest weight). Fish within the copy zone look for informed neighbors, whose swimming speed and direction they copy. Our default for this zone is the same as the alignment, assuming that from this distance the fish can clearly see changes in orientation.    Table), d. the distance at which the fish applies the copy-response. Figure S3.2 Sensitivity to rules weights. In our model, an agent in emergency mode is prioritizing cohesion and alignment over repulsion for the sake of avoiding separation from the group in strong attacks. In our tests, which were consisted of short local attacks, there is no difference in the speed of the flash wave between copy response with weights changes (left column) and copy response without weights changes (right column). Figure S4 shows frames from two samples: the flash signature caused by response to an attack on a cylindrical and a spherical school. The flash signatures of the two samples are clearly not identical, but it is hard to tell intuitively which of the differences are due to random factors in the sample (e.g. the location of the observer or noise in the orientation of the fish), and which are actually features characterizing the two different school shapes. These samples are used for training and testing our classifier.

Full results for the information content of shimmering waves
In this section we present the full results of the three models, each with and without the uncertainties (i.e. with or without roll-noise, lookat-moves, and observer-orientation. See 'Partial information and noise' section in the MS for details). For each model we present 'classification report (details in the figure below) as well as the confusion matrix.    6 Rationale of the selected models for differentiation 1. Attack direction: In case of a predator attack. We show that the flash signature of the school contains enough information to indicate that an attack has happened. We also show that it is possible to detect the direction of the attack. This indicates that the flashing signature could be a source of information in predator-prey dynamics.

Table of terms
2. School shape: We show that light pattern can be used to distinguish between school shapes. This proof-of-concept suggests that light patterns can be used to study the global, coarse-grained topology of the school (e.g. also density and orientation). To the best of our knowledge, the only current method to infer the shape and structure of large schools is based on the use of Eco sounders 3. Fish response: We show that light patterns could be used to distinguish among the four predatorevasion strategies (direct-only, direct-and-copy, flash-direct, and flash-latency). Success with this task will provide proof-of-concept that our method could aid in identifying the rules of collectivemotion during predator attack. In particular, it will provide indication of whether or not the fish themselves are using the flashes perceived from within the school to speed up the propagation of information.

Flash occlusions
The model described above does not take into account occlusions, i.e., cases in which rays are blocked by other fish. Here, we study a modified model in which occlusions are not ignored, demonstrating the results are qualitatively the same. To be precise, results with and without occlusions are quantitatively the same up to a scaling factor.
Occlusions were not considered in the measurement of the flash changes. We can get them graphically for a single (or a few) point(s), but calculating them for all fish would exceed a realistic workload. In S10 Fig we placed the camera at the back of the group of fish, let the predator attack, and for each pair of subsequent frames count the pixels with flash changes. I did this once with occlusions (painting the black fish) and once without. The model is of a copy behavior type without response to flashes. The occlusion version and the non-occlusion version are strongly correlated.
The area itself is also not ideal measurement since the fish may flash "in between" frames while its cross-section is different than the one presented. b. The flashes the observer sees without occlusions. c. The "real" image the observer sees.
Note that the red and the green are visual aids for us indicating whether the fish turning towards the observer or away of it. They are not "seen" by the observer. Also, the image here is not limited by the distance the observer "sees" when it calculates its Boids rules. d.
Correlation between the changes in total area of the flashes with and without occlusions in the different steps. Each plot (and dataset) was calculated on a different run but is replicable.
It is also important to note that in this simulation the fish are two -dimensional and in real three dimensional schools the effect of occlusion may be stronger and depend in the school density.

Results with the direct-only model
Here, we report additional simulation results establishing the absence of density or flash waves in the direct-only model without the copy response).

Sensitivity of copy and flash responses
Flash waves can be reproduced in both the direct-and-copy and flash-latency models. Here, we compare the two model as well as the flash-direct model, in particular the dependence on parameters.

11Robustness of the wave
In the MS we tested the waves on groups of a size of 15k where the Boids had a full viewing field and the attack came from the front. The escape response of the Boids were directed into the group and then away of the predator. We tested alternative scenarios where the attack came from the back and the Boids had a partial viewing field (a blind spot 30% of the back of the fish (Rountree and Sedberry 2009)). We also tested a scenario where the fish escape directly away of the predator without turning into the group first. In all cases we found that the wave exists and its speed is essentially the same in all configurations.
We also tested different angles. In order to do so, we let the observer to "surround" an attack scenario on a 5 k cylindrical school both horizontally (S17 Fig) and vertically (S18 Fig) and recorded the waves in 10 o differences. We found that except for "front view" of the school perpendicular to the wave direction, the wave pattern was detected in all angles. Interestingly, in the 'bottom view' (S19 Fig plots 240, 270, and 300 degrees) where all the fish are permanently reflecting light, we found an anti-flash wave.
Another interesting insight is that the wave speed was not constant from a diagonal horizontal view

Flash waves in shoaling groups
In order to imitate a shoaling (disordered) group, we took a school consisting of 5k fish and placed it in different random directions (S22 and S23 Figs). It turned out that the fish density and the flash signal towards the observer are less significant than for an aligned school, but can still be clearly seen. There is no clear flash change signal. The information-transfer signal (plot c. shows a clear wave of information). This can indicate that for a group of fish which is in a low level of alignment, the flash signal, as we defined it, is less useful for alarming about an upcoming event.

Flashes not due to evasive response
Not all the light reflection are caused by an evasive response of the fish. In this section, 'noise' is defined as the amount of visible flashes that are not caused by a reaction of the fish to a non-routine event, i.e. when the fish is obeying the regular Boids' rules and the school is keeping its direction.
Such noise can come from two sources: (i) Stable reflections when the 'standard location of the fish is on the reflection plane between the observer and the light source, and (ii) Random movement of the fish that are not caused by the Boids' rules.
Noise of type (i) can be caused when the fish is found 'under' a fish within the borders of the light distribution. I.e. if the fish stands 'straight' with its back fin pointed upwards and the light distribution is , an observer that is found within the range under the fish will see a reflection.
In particular, the observer that is placed in the center of a school sees a cone of reflecting fish on top of it. If the observer is keeping the same depth but is staying away of the school, it does not see any flashes. Intermediate distances from the school leads to intermediate results. Fig 24a. shows the number of stable reflections the observer sees when it moves horizontally and "travels through" a center of a school of 1000 fish. When the observer is away of the school it can see (almost) no reflections. When it gets close to the school, the top of the school starts reflecting light and in the middle of the school the fish in the cone at the size of the reflection angle on top of the fish are reflecting light. In this case, the maximal number of reflecting fish is ~100 which is slightly less than expected if the school would be round. This appears as logical since the school's shape is slightly oval.